-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support remotely-mountable layers for speeding up image distribution #644
Conversation
0f967c2
to
f8dea83
Compare
Based on the discussion in containers/podman#4739, I rethought the design of the remote snapshot functionality to leverage additional layer stores. What I've done here is to add a "layer discovery" functionality for the store. Layer discovery for additional layer storeCorresponding to the commit (smaller one): 7e66198 For some filesystems including stargz-based one, recognizing all available layers and storing the exhaustive list of I've made use of Working example of the graphdriver which supports remotely-mountable layersCorresponding to the commit (bigger one): f8dea83 As a working example, I implemented a stargz-snapshotter-based graphdriver and it's also included in this patch. This graphdriver leverages stargz snapshotter for mounting stargz layers and managing the raw contents and mountpoints. You can refer to the snapshotter implementation on https://github.com/ktock/stargz-snapshotter/tree/graphdriver. |
@ktock Could you fix the lint issues. Most of the engineering team has been tied up in working on Podman 2.0 we hope to get back to this soon. |
Thanks for your time. It seems to be Go version's issue. Do we have plan to move to a newer version of go(1.13)?
|
Yes, it is time we moved to golang 1.13 |
Signed-off-by: Kohei Tokunaga <[email protected]>
Signed-off-by: Kohei Tokunaga <[email protected]>
The completions of following tests haven't been reported to this PR page but tests (except for devicemapper ones) seem to have passed 🤔 https://cirrus-ci.com/build/5193639385104384
I think we can at first focus on the layer discovery part (commit 56035c0), then can proceed to the graphdriver implementation part (commit f98ab81). |
@@ -0,0 +1,993 @@ | |||
// +build linux |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there any code coming from containerd and we need to maintain the copyright header?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No.
And before I waste your time...
I don't intend to get graphdriver part (f98ab81) merged into libpod immediately. At this moment, it's just an example implementation and it's incomplete at some points,
- f98ab81 relies on stargz-snapshotter but it will require users to run additional daemons (=containerd+stargz-snapshotter) and it might not fit to the podman's design phirosophy?
- we might need a daemonless implementation of this patch.
- stargz-snapshotter is untested in the rootless world. (so the patch works only with root user)
- some implementation details are imcomplete yet (e.g. the way to treat id mappings, lockfiles, ...)
The patch f98ab81 intends to make sure that "layer discovery" functionality (56035c0) is needed to make remote-snapshotter-like functionality come true in podman.
I think the deeper review (or design discussion) of the graphdriver implementation can be after the "layer discovery" part (56035c0) is done (But it's kinda chicken and egg problem...).
Please tell me if I should separate the PR.
@ktock I wonder if this would be easier to implement adding a mechanism similar to additionalStores. Instead of expecting the store it is just the path to the exploded layers, as:
Or better, we could have it configurable and define a few substitution, e.g. Do you see any problem with this mechanism and CRFS? |
@giuseppe Querying layers to registries requires at least the image reference name (e.g. (Maybe in following-up patches => ) Furthermore, we'll need a verification functionality for each lazily-fetched chunk (stargz-snapshotter is starting to support it). The OCI digest of a layer is required for querying it on the registry but can't be used for verifying each lazily-fetched file data so we'll need to pass additional information (e.g. the OCI digest of stargz TOC contained in the manifest as a layer annotation) to the underlying filesystem, e.g. |
Why base64 encoded? To avoid slashes in the name? In our experience the complex part is not really having the layers accessible, but to create all the accessory file system structure next to it, like the layer.json file and the whole directory structure necessary. If we could get along with just providing the layers in the filesystem it would be great! Info gather by our GSoC student: https://medium.com/@mohit2501tyagi/lets-walk-through-podman-37636cf223c5 |
Closing in favor of containers/podman#8837, #795 and containers/image#1109 . |
Related:
The following design description is stale. See #644 (comment) for the latest design of this patch.
Recently I read through the codes and came up with the strategy for enabling remote-snapshotter-like functionality in libpod. This is a discussion thread for the design of the low-level part of this functionality.
Creating layers without pulling the content means that
Store.ApplyDiff
API won't be called for this layer. So during the call forStore.CreateLayer
API, we need to check the existence of the targeting layer content in the graphdriver. If the content exists we need to commit it immediately like done in ApplyDiff.Creating layers without ApplyDiff
Corresponding to the commit (smaller one): d174476
To achieve this,
storage.Store
will need an additional calling convention with graphdrivers for asking if the targeting layer contents can be provided by graphdrivers (i.e. remotely-mountable). In this patch, I extended the semantics ofDriver.Create
API. As mentioned in another PR(...), additional information for that layer is passed fromImageDestination
toStore.CreateLayer
API usingstorage.LayerOptions.Labels
option. ThenStore.CreateLayer
passes these labels toDriver.Create
API usingdriver.CreateOpts.StorageOpt
, for asking if the targeting layer is remotely-mountable.The graphdrivers (e.g. CVMFS-graphdriver or stargz-graphdriver, etc) can search the remote mount points using passed
StorageOpts
. If the layer is remotely mountable, graphdrivers tell so to the store. In this patch, I introduced a typed errorErrTargetLayerAlreadyExist
for this. When the store getsErrTargetLayerAlreadyExist
, it immediately registers this layer in the same way as done inStore.ApplyDiff
API.Working example of the graphdriver which supports remotely-mountable layers
Corresponding to the commit (bigger one): b139d87
As a working example, I implemented a stargz-snapshotter-based graphdriver and it's also included in this patch. This graphdriver leverages stargz snapshotter for mounting stargz layers and managing the raw contents and mountpoints. You can refer to the snapshotter implementation on https://github.com/ktock/stargz-snapshotter/tree/graphdriver.
TODOs